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Perhaps  one  of  the  most  striking  findings  in  the  area  of  human  decision 
making  is  that  of  conservatism.  Edwards  (1968)  summarizes  the  phenomenon: 

An  abundance  of  research  has  shown  that  human  beings  are 
conservative  processors  of  fallible  information.  Such  ex¬ 
periments  compare  human  behavior  with  the  outputs  of  Eayes's 
theorem,  the  formally  optimal  rule  about  how  opinions  (that 
is,  probabilities)  should  be  revised  on  the  basis  of  new 
information.  It  turns  out  that  opinion  change  is  very  or¬ 
derly,  and  usually  proportional  to  numbers  calculated  from 
Bayes's  theorem— but  it  is  insufficient  in  amount  (pp.  17-18). 

In  many  settings  it  is  extremely  desirable  to  extract  from  the  information  at 
hand  all  of  the  certainty  that  is  possible,  not  only  because  it  is  important 
to  make  as  informed  a  decision  as  possible  but  also  because  information  can  be 
very  expensive.  Notable  areas  where  these  considerations  apply  are  military 
intelligence^  medical  diagnosis,  business  decision  making,  etc. 

Why  are  men  conservative  information  processors?  The  contributing  factors 
are  several  (see  DuCharme,  1969;  and  Edwards,  1968,  for  reviews  of  relevant  re¬ 
search)  but  the  outstanding  one  seems  to  be  misaggregation  of  information. 
People  seem  to  assess  correctly  the  impact  of  a  single  datum,  but  do  not  com¬ 
bine  data  correctly.  Edwards  has  proposed  a  probabilistic  information  process¬ 
ing  system  to  overcome  human  suboptimality  (Edwards,  1962;  Edwards,  Lindman  & 
Phillips,  1965}  Edwards,  Phillips,  Hays  &  Goodman,  1968). 


In  their  abstract,  Edwards,  Phillips,  Hays  &  Goodman  (1968)  summarized 
this  system: 

A  Probabilistic  Information  Processing  System  (PIP)  us°s  men 
and  machines  in  a  novel  way  to  perform  diagnostic  informa¬ 
tion  processing.  Men  estimate  likelihood  ratios  for  each 
datum  and  each  pair  of  hypotheses  under  consideration  or  a 
sufficient  subset  of  these  pairs.  A  computer  aggregates 
these  estimates  by  means  of  Rayes's  theorem  of  probabil¬ 
ity  theory  into  a  posterior  distribution  that  reflects  the 
impact  of  all  available  data  on  all  hypotheses  being  con¬ 
sidered.  Such  a  system  circumvents  human  conservatism  in 
information  processing,  the  inability  of  men  to  aggregate 
information  in  such  a  way  as  to  modify  their  opinions  as 
much  as  the  available  data  Justify. 

Edwards,  et  al.,  go  on  to  state  their  major  empirical  results:  data  which  lead 
PIP  to  give  99:1  odds  favoring  a  hypothesis  resulted  in  4.5:1  odds  in  favor  of 
that  hypothesis  when  the  data  were  evaluated  by  a  non-PIP  system.  They  inter¬ 
pret  this  result  as  demonstrating  that  PIP  used  the  information  contained  in 
the  data  much  more  efficiently  than  do  competing  inference  systems  they  employed 
in  their  simulation.  The  implication  of  this  superior  efficiency  is  that  in 
situations  where  many  pieces  of  information  must  be  processed  it  should  be  done 
by  a  PIP- like  system  and  that  system's  output  should  be  the  basis  of  decision. 
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One  common  attribute  of  settings  where  It  may  be  profitable  to  employ  a 
PIP  system  for  Information  aggregation  Is  that  It  Is  very  important  to  make  the 
best  decision  possible.  Thus  the  fact  that  a  preponderance  of  the  evidence 
points  In  one  direction  may  not  be  as  Important  as  the  exact  odds  level  associ¬ 
ated  with  the  evidence.  Ihe  difference  between  being  60:1  certain  and  being 

99:1  certain  may  be  very  large  In  terms  of  the  course  of  action  that  Is  decided 
upon. 


in  the  Edwards,  et  al.,  simulation,  PIP  was  the  only  processing  system 
which  reached  these  levels  of  certainty  from  the  data.  But  their  simulation 
concerned  complex  hypothetical  scenarios  of  the  "world  of  1975"  which  did  not 
allow  the  possibility  of  calculating  "correct"  odds.  PIP  was  certainly  non¬ 
conservative  compared  with  the  other  systems,  but  there  is  no  assurance  that 
PIP  may  not  have  been  too  extreme  i.e.,  ended  up  more  certain  than  the  data 
Justified.  Edwards,  et  al.,  cite  an  experiment  by  Phillips  (1966)  which  em¬ 
ployed  a  PIP  system  in  a  situation  where  "correct"  odds  could  be  externally 
calculated.  He  found  a  resulting  difference  in  processing  systems  comparable 
to  the  Edwards,  et  al.,  finding  and  furthermore  reported  that  PIP  itself  ap¬ 
peared  somewhat  conservative.  However,  in  her  dissertation,  Wheeler  (1972) 
examined  mis  aggregation  as  the  source  of  conservatism.  She  used  a  data  gener¬ 
ating  source  for  which  the  "correct"  odds  could  be  calculated  and  the  results 
strongly  point  to  missagregation  as  the  basic  source  of  human  conservatism. 
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She  reports  very  conservative  posterior  odds  associated  with  hi.mnn-aggregated 
information  and  nearly  veridical  posterior  odds  associated  with  machine-aggre¬ 
gated  information.  However,  in  the  experimental  condition  where  the  data  were 
relatively  undiagnostic  the  machine-aggregated  posterior  odds  were  somewhat  ex¬ 
cessive  when  compared  with  the  veridical  posterior  odds  (median  machine-aggre¬ 
gated  log  odds  VD.  veridical  log  odds  showed  correlation  of  .967  and  regression 
slope  of  1.514). 

But  it  is  in  exactly  this  sort  of  setting,  where  there  is  an  abundance  of 
relatively  undiagnostic  data,  that  PIP  is  most  likely  to  be  applied  (e.g.,  mil¬ 
itary  intelligence).  It  becomes  evident  that  the  development  of  an  error  theory 
for  PIP  is  extremely  desirable.  The  following  discussion  attempts  to  extend 
present  knowledge  in  the  direction  of  such  a  theory. 

Bayes's  theorem  is  the  formally  optimal  tool  for  revising  probabilistic 
opinion  in  the  light  of  new  information,  (liven  a  hypothesis  and  a  datum  bear¬ 
ing  on  that  hypothesis,  Bayes's  theorem  states  that: 

PtnlB,  -  iMilpl  .  | 

Here,  D  represents  the  datum,  H  the  hypothesis,  P(d|h)  is  the  probability  of 
the  datum  given  the  truth  of  the  hypothesis,  P(h)  the  prior  probability  of  the 
hypothesis,  r(r)  the  probability  of  observing  the  datum  (unconditionally),  and 
r(H[p)  the  posterior  probability  of  the  truth  of  the  hypothesis.  For  two  com- 
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P^ting  hypotheses,  H^,  Hp,  and  a  datum,  D,  bearing  on  them,  Bayes's  theorem  may 
be  used  in  odds  form: 

PCHjD)  p(d|h1)p(h1) 

P(Hg|D)  "  P(D|Hp)P(H?) 
or, 

=  L  •  fi1, 

where  is  the  prior  odds  for  U±  against  Hp,  and  L  is  the  likelihood  ratio  as¬ 
sociated  with  D.  The  quantity  L  completely  summarizes  all  of  the  diagnostic 
impact  of  D  vis  a  vis  and  H?  (see  Edwards,  Lindman  &  Savage,  1963).  When  a 
sequence  of  conditionally  independent  data  are  considered,  Bayes's  theorem  may 
be  applied  iteratively,  the  posterior  odds  for  the  (n  -  l)th  datum  being  used  as 
the  prior  odds  for  the  incorporation  of  the  n  th  datum. 

In  a  PIP  system  human  experts  estimate  the  relative  impact  of  each  datum, 
i.e.,  they  estimate  the  quantity  L,  and  a  computer  aggregates  these  according 

to  Bayes's  theorem.  The  computer,  which  is  just  Bayes's  theorem  mechanized, 

never  observes  the  basic  datum-— it  receives  only  an  operator's  estimate  of  the 

impact  of  the  datum,  the  likelihood  ratio.  Thus  to  the  computing  system,  the 

basic  datum  is  the  operator's  estimate  of  the  likelihood  ratio. 

Men  seem  very  adept  at  estimating  likelihood  ratios  (much  of  the  litera¬ 
ture  establishing  thip  is  reviewed  in  Edward_,  1968).  However,  if  error  enters 
into  the  PIP  output  it  enters  a  this  point— where  the  human  operators  trans¬ 
form  the  information  contained  in  the  data  into  numbers.  Even  if  the  estima¬ 
tors  are  very  good  on  the  average  in  assessing  the  diagnostic  impact  of  the 
data,  their  estimates  are  bound  to  be  somewhat  variable.  That  is,  if  on  some 
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occasion  an  operator  estimates  the  likelihood  ratio  for  some  datum  to  be  1.7 
we  cannot  be  sure  that  at  some  other  time  for  that  same  datum  and  situation  he 
might  have  estimated  the  likelihood  ratio  to  be  a  little  more  or  a  little  less 
than  1.7.  Unfortunately  the  formal  nature  of  probabilist ie  inference  is  such 
that  this  sort  of  variability  doesn't  tost  "cancel  out."  When  additional  vari¬ 
ability  is  introduced  into  an  inference  system,  the  inevitable  result  is  to 
limit  the  level  of  certainty  which  that  system  can  reach.  In  some  cases  the 
effect  of  this  sort  of  source  unreliability  can  be  quite  surprising  (Schum  & 
DuCharme,  1969)* 

What  does  this  mean  for  a  PIP  system  which  uses  human  estimates  of  likli- 
hood  ratios?  When  PIP  receives  a  likelihood  ratio  to  be  entered  into  the  com¬ 
puting  algorithm,  this  ratio  can  be  taken  to  mean:  "a  datum  with  this  exact 
diagnostic  value  has  occur-ed."  This  is  the  definition  adopted  by  Edwards,  et 
al.  (1968).  Given  this  interpretation,  the  report  of  a  likelihood  ratio  is  Just 
as  diagnostic  as  the  observation  of  the  datum. 

A  second  interpretation  is  suggested  by  consideration  of  human  variability. 
Here,  when  a  likelihood  ratio  is  given  to  the  computer  it  is  interpreted  as  "a 
datum  has  been  observed  and  its  diagnostic  value  is  estimated  to  be  (some  num¬ 
ber).  Use  this  estimate  to  infer  the  true  diagnosticity  of  the  datum  and  use 
the  result  of  that  inference  as  input  to  inference  algorithms  for  and  H  . " 
Note  that  adopting  this  second  interpretation  introduces  into  the  inference 
problem  uncertainty  based  on  the  quality  of  the  estimated  likelihood  ratios. 

Any  inference-making  system  which  takes  this  datum-level  uncertainty  into 
account  can  never  be  as  sure  of  itself  ( so  to  speak)  as  a  system  which  ignores 
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this  source  of  uncertainty— as  does  the  PIP  used  by  Edwards,  et  al.  Maybe  PIP 
was  extracting  more  certainty  than  the  data,  i.e.,  the  estimates,  justify. 

Thus  the  goal  of  an  error  theory  for  PIP,  as  PIP  is  currently  conceptualized, 
is  to  characterize  the  source  of  possible  error  in  the  system  and  to  incorpor¬ 
ate  this  characterization  into  the  inference  algorithm  in  a  formally  optimal 
manner.  An  outline  for  one  approach  to  this  problem  comprises  the  rest  of  this 
paper. 

Suppose  that  our  basic  problem  is  deciding  between  two  competing  hypoth¬ 
eses,  H  and  F  .  As  the  basis  for  inference  we  collect  data,  d,  ,d  ,...,d  el), 
12  1  2  n 

where  D  is  the  set  of  all  data  relevant  to  the  two  hypotheses.  The  diagnostic 
impact  of  a  datum,  d^,  is  completely  described  by  its  likelihood  ratio, 

For  each  d^  there  exists  a  L(d^)  e'J  ,  where 

P(dt|H.) 

L(di)  =  P(di|Hp)  ’ 

and-^is  the  set  of  positive  real  numbers.  Since  the  map  L:D  Vsf  is  manyrl,  it 
makes  sense  to  spe'ik  of  ;t  density  function  over  the  setw^under  the  function 
L(P). 

If  the  density  function  f( d |hi )  /  f(d|H?),  then  it  is  possible  to  speak  of 
two  distinct  density  functions  overvt  ,  and  f(L|H?).  Not,  speaking  of 


p(^|h2) 


I 


the  basic  datum  fcr  Hayes's  theorem  as  a  likelihood  ratio,  it  is  perfectly  per¬ 
missible  to  write: 

pChJl)  p(l|h1)  p(h1) 

p(H2|l)  p(  J *  I H0 )  ‘  p(  mo)  * 

Given  an  L  sampled  from  the  distribution  with  density  f(i|H  ),  the  estimated 
likelihood  ratio  can  be  characterized  as  I,' ,  a  random  variable  with  density 
f'(L'|L),  where  the  distribution  of  the  estimated  likelihood  ratio  is  dependent 
only  upon  the  value  of  the  veridical  likelihood  ratio  (this  assumes  that  L'  is 
conditionally  independent  of  H;  i.e.,  f(L' |L,H)  -  f(L* |l)).  Now,  following  the 
development  by  Gettys  &  Willke  (1969),  we  may  write 

p(i.’|hJf(h  ) 


where 

P(L'|H  )  =  /  f'(L'|L)f(L|H  )dL  . 

(Gettys  and  Willke  use  summations;  here  the  continuous  case  is  considered,  in 
either  case  the  arguments  are  parallel.) 

Then  in  odds  form  this  becomes 


P(H1|L')  P(H1)  /^'(L’  |L)f(L|H1)dL 

P(H?|L')  =  P(Hp)  '  /^f,(L’|L)f(L|H?)dL 
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Taking  logs,  we  get 

p(HjJl*  )  pfty  ^'(L’lMfdlHjJai 

log  P(H2|L')  =  l0B  ioy  *  108  (s' 

The  derivation  of  equation  (l)  depends  on  three  r  mptions.  These  may 
be  itemized  as: 

(Al)  L'  (the  estimated  likelihood  ratio)  is  conditionally  inde¬ 
pendent  of  and  of  H^. 

(A?)  f '(L |l^)  and  f'(L'|L  )  differ  only  in  the  value  of  the  pa¬ 
rameters  L  for  all  i,  j. 

(A3)  the  value  of  the  variable  L'  is  known. 

To  actually  calculate  the  posterior  odds,  given  an  estimated  likelihood  ratio, 
one  must  specify  the  conditional  probability  density  functions  ^Lf^),  f(L|H2), 
and  f'(L'|L).  The  importance  of  the  vprious  parameters  which  these  functions 
involve  to  the  form  of  the  ratio  of  integrals  in  equation  ( l)  is  demonstrated 
by  the  following  example. 

Assume  that  the  basic  datum  input  to  the  PIP  system  is  the  logarithm  of  a 
likelihood  ratio  estimate.  Let  L  be  the  log  veridical  likelihood  ratio,  and  1' 
be  the  log  estimated  likelihood  ratio.  Now,  under  the  log  transformation,^  if 

redefined  the  set  of  all  real  numbers.  Assume  that  f(L|H  )  is  the  normal 

2 

density  with  parameters  u  and  o^  Note  that  by  specifying  f(L|H  )  the  density 
f(LlH2)  is  automatically  specified  because  the  relation 
f(L|H1) 

?<r[ig  -  exp(L)  (5) 
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must  be  satisfied  at  all  points  L  eX>(here  we  assume  that  logarithms  to  the  base 

e  are  used).  Assume  further  that  the  density  f'(L'|L)  is  also  the  normal  den- 

2 

sity  function,  with  parameters  L  and  a  . 

Ii 

In  words,  it  has  been  assumed  that  if  H  were  true  and  we  collected  all 
possible  data  that  would  distinguish  between  and  H  and  observed  the  distri¬ 
bution  of  the  veridical  log  likelihood  ratios  associated  with  the  data  we  would 

find  that  it  was  the  normal  probability  distribution  with  mean  p  and  variance 
2 

ah*  Furthermore>  given  a  particular  datum  with  log  likelihood  ratio  L  the  es¬ 
timate  which  would  be  given  to  the  PIP  system,  L',  may  be  thought  of  as 

L'  =  L  +  os,  where  m  is  a  random  error  which  is  normally  distributed  with  a 

2 

mean  of  zero  and  variance  In  other  words,  the  estimator  is  pretty  good  in 

that  on  the  average  when  a  datum  of  diagnosticity  L  occurs  he  will  give  an  es¬ 
timated  diagnosticity  L*  =  L,  but  may  vary  around  the  true  value  a  little  bit. 

The  quantity  of  interest  Is  the  likelihood  ratio  that  should  be  used  in 
Bayes's  theorem  when  the  estimator  gives  and  estimate  of  L'.  This  quantity  is 
the  ratio  of  integrals  in  equation  (1).  Substituting  into  the  numerator  of  this 
ratio  the  densities  that  have  been  assumed  we  get: 


J  f’(L'|L)f(L|H1)dL  =  _C~~  exp 


yfZ  2 

2rtcr 

Ii 


1,  L'  -  L.2 


2  a. 


r 


\T  2 

2rtcr, 


exp 


h 


Iri  the  denominator,  using  the  assumptions  and  relationship  (3),  we  get: 


^f’(L'L)f(L|H?)dL  =  [l 


2U  L 


)  1/ L'  -  L,2 

exp^  ( - ) 

l  ; 


^__expf.^)2.LfaL 

2  no. 


Simplifying  and  taking  logarithms  we  get  the  adjusted  log  likelihood  ratio: 

2  2 

£f'(L'|L)f(L|H  )dL  L ’o2+»o?-\± 

T  ii  _  i  ^ _ 1  _ h  L  2 

6  f  f’(L'|L)f(L|H  JdL  "22  (  4) 

>*-  2  a  +  a 

h  L 

This  is  the  quartity  that  is  used  in  (2),  the  log  odds  form  of  Bayes's  theorem. 
The  quantity  that  the  Edwards,  et  al.  (1968)  study  used  to  calculate  posterior 
log  odds  would  have  been  V ,  whereas  if  variability  in  likelihood  ratio  esti¬ 
mates  is  taken  into  account  the  quantity  that  would  have  been  used  is  L"  which 

is  somewhat  less  than  L'.  Note  that  L"  =  L'  if  there  is  no  possibility  of 

2 

error  in  the  estimates,  i.e.,  =  0.  How  much  less  than  L'  is  L"  depends  upon 

the  parameters  ah,  a^,  and  y.  under  the  assumptions  that  have  been  made. 

To  obtain  some  idea  of  the  magnitude  of  the  adjustment  from  L'  to  L",  con¬ 
sider  a  numerical  example.  In  the  Edwards,  et  al. ,  study  the  posterior  odds  of 
99:1  were  achieved  by  the  cumulative  impact  of  60  data.  Since  the  particular 
hypothesis  being  considered  began  with  odds  of  1:5,  the  posterior  odds  repre¬ 
sents  a  cumulative  likelihood  ratio  of  495,  or,  in  natural  logarithms,  6.204. 
The  average  log  likelihood  ratio  estimate  was  thus  approximately  .103.  Since 
the  model  of  estimator  error  assumes  that  the  average  estimate  if,  the  veridical 
value,  let  us  assume  for  the  sake  of  example  that  u  =  .103.  The  data  reported 
by  Edwards,  et  al. ,  do  not  allow  calculation  of  the  variance  of  log  likelihood 
ratios  for  the  60  data  scenario  leading  to  the  posterior  odds  of  99:1.  However, 
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they  do  give  a  distribution  of  likelihood  ratios  for  all  data  used  in  all  scen¬ 
arios  (some  5400  data).  The  overall  variance  may  be  approximated  from  the  fig¬ 


ures  they  give  (page  862,  Table  V);  using  this  yields  the  estimate  CT?  =  .0253. 

h 

In  practice,  to  use  equation  (4)  one  would  obtain  a  log  likelihood  ratio 
estimate,  L',  for  each  datum  then  solve  for  L",  the  adjusted  log  likelihood 
ratio,  and  use  this  value  in  equation  (2)  to  revise  the  prior  log  odds.  For  the 
Edwards ,  e_t  al. ,  data  we  do  not  have  ohe  individual  likelihood  ratio  estimates 
for  the  60  data  sequence.  Instead,  for  the  sake  of  example,  we  will  use  equa¬ 
tion  (4)  to  obtain  L"  corresponding  to  an  estimate  of  L'  =  .103,  the  mean  for 
the  6o  data.  Multiplying  the  resulting  L"  by  6o  gives  us  the  expected  log  cum- 
mulative  likelihood  ratio  for  the  6o  data  taken  together.  This  value  is  then 
used  to  give  a  rough  estimate  of  what  the  posterior  odds  might  have  been  for 
Edwards,  et  al.,  data  sequence  if  estimator  variability  had  been  taken  into 
account.  The  following  table  displays  these  calculations  for  selected  levels 

of  of. 

L 


Posterior  Adjusted  Odds  for  Selected  Levels  of 


o 


2 

L 


2 

°L 

L" 

Estimated 

Posterior  Odds 

.0000 

.1030 

99.00 

.0005 

.1028 

92.19 

.0010 

.1025 

95.86 

.0015 

.1023 

9?.50 

.0020 

.1021 

91.38 

.0025 

.1019 

90.23 

.00?0 

.1017 

80.14 

.0035 

.1015 

88.09 

.0040 

.1013 

87.10 

.0045 

.1011 

86.14 

.0050 

.1009 

85.23 

2 

O 

T." 

Estimated 

L 

Li 

Posterior  Odds 

,0055 

.1007 

84.36 

0060 

.1006 

83.52 

OO65 

.1004 

82.72 

0070 

.1003 

81.95 

0075 

.1001 

81.21 

0080 

.1000 

80.50 

OO85 

.0998 

79.82 

0090 

.0997 

79.16 

0095 

•0995 

78.53 

0100 

.0994 

77.91 

[ 

i 

* 

] 


0 

0 

D 

li 

L 

b 

U 

D 

0 

0 

0 


As  may  be  seen  from  the  calculations,  the  effect  of  estimator  variability 
on  the  posterior  odds  may  be  quite  slight  in  this  situation  where  there  is  an 
abundance  of  data  which  is  relatively  slight  in  its  diagnostic  -luo.  A  reason¬ 
able  question  is  what  values  of  one  might  encounter.  Unfortunately  the  lit¬ 
erature  on  likelinood  ratio  estimation  is  rarely  reported  in  a  way  that  allows 

this  quantity  to  be  estimated.  Edwards,  et  al.,  do  report  standard  deviations 

2 

for  repeated  estimates  by  their  subjects  which  may  be  used  to  estimate  a^. 

Their  value  averaged  around  .0007*  From  the  table  it  may  be  seen  that  this 
amount  of  variability  would  have  only  minimal  effect  on  the  reported  posterior 
odds.  Wheeler  (1972),  whose  subjects  gave  estimates  for  abstract  laboratory 
stimuli,  does  not  report  a  number  comparable  to  but  inspection  of  her  Fig.- 
ure  5,  a  plot  of  veridical  log  likelihood  ratios  against  median  estimated  log 
likelihood  ratios,  allows  us  to  take  a  rough  guess  that  a  value  of  .01  may  not 
be  unreasonable  in  the  range  of  ±  .4  for  the  veridical  log  likelihood  ratios. 
This  size  variability  would  decrease  the  PIP  posterior  odds  from  99  to  about  78. 

It  must  be  emphasized  that  the  calculations  that  the  table  is  based  upon 
employ  only  the  roughest  estimates  of  the  parameters  needed  for  exact  solutions. 
In  addition,  the  model  of  man  as  an  estimator  that  is  implicit  in  the  equations 
is  a  fairly  generous  one.  It  assumes  that  the  only  source  of  estimate  error  is 
random  variability.  This  may  not  be  the  case  at  all— more  likely,  variability 
increases  with  the  magnitude  of  the  veridical  log  likelihood  ratio  (see  Wheeler, 
197?,  Figure  5)»  However,  as  has  been  pointed  out  above,  situations  where 
PIP  might  be  applied  seem  most  often  to  be  those  where  the  data  is  of  low  diag- 
nosticity,  i.e.,  low  average  log  likelihood  ratio  and  not  too  much  variability 
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wound  that  average.  The  example  ueed  for  the  table  may  be  considered 
prototypical  of  this  type  of  setting. 

One  possible  criticism  of  the  present  development  concerns  the  distribu¬ 
tional  assumptions  that  have  been  made.  The  theory  which  culminated  in  equa¬ 
tion  (2)  is  sound  regardless  of  the  true  data  and  estimator  distributions. 
However,  one  may  argue  with  the  further  development  which  assumes  that  ffL|H) 
and  f'(L'|L)are  Oaussian  densities .  We  would  argue  that  for  the  sake  of  study¬ 
ing  analytically  the  effect  of  estimator  variability  on  the  PIP  posterior  odds 
that  the  forms  assumed  for  these  densities  should  probably  serve  very  well  as 
approximations  in  the  settings  being  considered.  Unless  the  data  generating 
process  is  very  perverse  the  distribution  of  veridical  log  likelihood  ratios 
will  be  roughly  single  peaked.  Once  this  condition  is  met  the  normal  density 
can  be  considered  to  be  a  very  good  first  approximation.  If  one  is  very  con¬ 
cerned  about  the  fit  (or  lack  of  it)  he  can  always  be  generous  in  estimating 
the  variance,  since  the  magnitude  of  the  adjustment  in  log  likelihood  ratio  in¬ 
creases  monotonically  with  variance  of  the  normal  distribution  used  to  approxi¬ 
mate  f(L|H).  The  result  will  be  n  conservative  estimate  of  the  adjusted  posterior 
odds. 

Assuming  f '(L1  |L)  to  be  the  normal  density  may  not  be  a  bad  approximation 
at  all.  Thi6  assumption  is  quite  pervasive  in  the  area  of  psychophysics.  One 
should  worry  more  about  assuming  that  the  variance  is  independent  of  L  than 
assuming  the  f'  to  belong  to  the  family  of  normals.  Here  again,  though,  inad¬ 
equacy  of  fit  may  be  compensated  for  by  overestimating  the  variance  which  in 
turn  overestimates  the  necessary  adjustment  to  the  PIP  posterior  ^dds. 


The  steps  one  would  go  through  to  apply  the  present  error  theory  in  an 
applied  PIP  situation  follow  two  distinct  paths.  Tf  the  necessary  distributions 
are  known  the  theory  could  he  applied  at  the  L  vel  of  the  computing  algorithm. 
This  will  almost  never  be  the  case.  Assuming  that  the  requisite  distributions 
are  unknown,  one  could  make  assumptions  similar  to  those  made  here  and  compute, 
using  equation  ( 4)  iteratively,  datum  by  datum.  Alternatively  one  could  go 
ahead  as  did  Edwards,  e_t  al. ,  and  use  the  estimates  as  if  they  were  veridical; 
then,  using  assumptions  we  have  outlined,  retrospectively  examine  i'ne  numerical 
consequences  of  assuming  various  amounts  of  variability  in  the  estimates  as  we 
did  in  the  table 

This  latter  route  would  be  by  far  the  easiest  because  it  is  highly  likely 

2  2 

that  the  exact  parametric  values  for  u,  a  ,  and  a  needed  for  the  datum  by  datum 

h  L 

application  can  only  be  estimated  in  retrospect.  Presumably  the  posterior  odds 
level  at  which  the  course  of  action  to  be  chosen  will  change  is  known  before  the 
data  are  evaluated.  The  retrospective  application  of  the  error  theory  as  out- 

O 

lined  above  will  show  just  how  much  error,  i.e.,  the  magnitude  of  a  ,  can  be 
tolerated  without  changing  the  decision.  If  even  generous  estimates  of  esti¬ 
mator  variability  fall  within  this  region  of  tolerance,  then  we  can  feel  assured 
that  inherent  variability  will  not  be  deleterious  to  the  decision.  On  the  other 
hand,  if  likely  levels  of  variability  will  possibly  change  the  decision,  one 
would  be  advised  to  recompute  on  a  datum  by  datum  basis  using  the  retrospective 
parametric  estimates  and  use  the  resulting  adjusted  posterior  odds  as  the  basis 
for  decision  or  to  reserve  judgment  until  more  conclusive  evidence  is  collected. 


Reproduced  from 
besf  available  copy. 


This  last  consideration,  what  to  do  if  the  error  analysis  indicates  the 
decision  could  be  affected,  points  up  the  major  weakness  of  the  present  devel¬ 
opment.  In  practice  there  will  be  formidable  obstacles  to  acquiring  the  neces¬ 
sary  knowledge  to  apply  the  theory  as  an  integral  part  of  the  aggregation 
algorithm.  The  true  distributions  cr.d  their  necessary  parameters  will  almost 

never  be  known  making  it  impossible  to  apply  equation  (2)  in  exact  form.  When 
(2)  is  applied  in  approximate  form  as  demonstrated  here,  we  may  be  conclusive 
in  saying  when  the  PIP  posteriors  are  sufficient  for  making  a  given  terminal 
decision,  but  not  in  saying  when  they  are  not  sufficient.  The  major  advantage 
of  the  present  theory  is  that  it  uses  log  likelihood  ratio  estimates  as  its 
basic  input  rather  than  individual  likelihoods  or  probabilities,  the  latter  task 
being  one  at  which  humans  are  far  less  adept  (Phillips  &  Edwards,  1966)  Other 
discussions  of  datum  reliability  are  closely  tied  to  this  later  type  of  judgement 
(see  Kelly,  1972). 

As  a  final  note,  it  should  be  pointed  out  that  the  alternatives  to  PIP  are 
as  subject  to  estimator  varii  bility  as  is  PIP.  The  next  best  system  for  infor¬ 
mation  processing  examined  by  the  Edwards,  el  al.,  study  was  christened  POP. 

In  this  mode,  operators  reestimated  the  actual  posterior  odds  after  seeing  the 
data  instead  of  estimating  likelihood  ratios  and  letting  the  computer  aggregate 
for  them.  When  PIP  was  reporting  odds  of  99:1,  POP  was  giving  4.5:1.  But  again, 
if  we  were  to  take  into  account  inherent  variability  in  the  POP  posterior  odds 
estimates,  the  4.5:1  odds  would  be  reduced  as  are  the  PIP  odds.  The  same  basic 
approach  used  here  would  load  to  a  similar  error  theory  for  POP,  however  some 


-  16  - 


of  the  algebra  would  be  different.  Such  a  development  is  beyond  the  scope  of 
the  present  paper. 

Where  to  go  from  here?  The  theory  that  has  been  developed  is  sufficiently 
promising  to  warrant  more  intensive  work,  Tbe  goals  of  further  research  should 
include  both  extension  and  refinement  of  the  theory  and  broadening  the  empirical 

basis  for  application.  The  first  obvious  extension  of  the  theory  is  obtaining 
concise  formulas  for  adjusted  likelihood  ratios  suet  as  equation  (4),  but 

basing  the  derivation  on  distributional  assumptions  other  than  the  ones  used 
here.  Once  concise  forms  are  obtained,  there  should  be  numerical  work  done  to 
examine  the  magnitude  of  error  possibly  introduced  by  using  the  wrong  assump¬ 
tions  as  approximations.  We  expect  that  most  reasonable  distributions  will 
not  significantly  change  the  conclusions  we  have  reached  using  the  normal  ap¬ 
proximations;  but  it  would  be  nice  to  have  the  numerical  work  to  support  this 
statement. 

For  empirical  work,  the  first  step  would  be  to  characterize  the  distribu¬ 
tions  of  data  likelihoods  that  may  be  encountered  in  application  of  PIP  in  a 
variety  of  information  processing  settings.  A  concurrent  line  of  investiga¬ 
tion  would  comprise  building  an  empirical  data  base  from  which  the  f'(L' |l) 
distributions  could  be  obtained.  One  byproduct  of  this  would  be  a  characteri¬ 
zation  of  those  inference  problems  in  which  estimator  variability  is  sufficient 
to  warrant  applying  the  theory  on  a  datum  by  datum  basis  rather  than  the  retro¬ 
spective  fashion  we  have  described. 
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